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ABSTRACT 



Research has shown that multiple-choice questions 



formed by transforming or paraphrasing a reading passage provide a 
measure of student comprehension. It is argued that similar 
transformation and paraphrasing of lesson questions is an appropriate 
way to form parallel multiple-choice items to be used as a posttest 
measure of student comprehension. Four parallel items uiay be derived 
from a lesson: (1) an item that is neither transformed nor 
paraphrased, thus testing simple rote memory; (2) an item that is 
transformed only; (3) an item that is paraphrased only; and (4) a 
compound item that is both transformed and paraphrased. These four 
Items produce a hierarchy of difficulty from easy (rote) to difficult 
(compound) , depending on the extent to which an item is altered from 
the original lesson question. In the three studies cited in this 
paper, posttest performance for the four types of items followed the 
hypothesized hierarchy of difficulty, and this is viewed as evidence 
that many of the cognitive skills underlying comprehension may also 
be hierarchically related. It is concluded that the formation of test 
questions from lesson items provides a practical way for designers to 
utilize multiple-choice items for instruction and also for posttest 
assessment. (DB) 
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Modifying Multiple-Choice Questions in Computc;r-Based Instruction 

October 1990 
Abstract 

This paper describes two methods for adapting multiple-choice questions in 
order to form parallel items at different levels of comprehension. Following Bormuth 
(1970), Anderson (1972) described a way to form posttest questions (i.e. items) from 
reading passages. He argues that these items formed by transforming or paraphrasing 
a passage provide a measure of student comprehension of the passage. Applying this 
same method to lesson items appears to be an i^prc^riate way to form parallel posttest 
items. In the three studies cited, posttest performance for the four types of items 
followed the hypothesized hierarchy of difficulty. These results are viewed as 
evidence that many of the cognitive skills underlying comprehension "may be 
hierarchically related" (Bormuth, Manning, Carr, & Pearson, 1970). This 
methodology appears to provide a differential measure of comprehension. If so, this 
method provides a practical way foe designers to utilize multiple-choice items for 
instruction and also for posttest assessment. 
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Modifying Multiple-Cho^''^. Questions in Computcr-Based Instruction 

October 1990 

Multiple-choice questions (i.e. items) are an unusually convenient and powerful 
instructional f(Mtnat in computer-based lessons that provide information to the learner 
through the stem, correct response option, and incorrect options. They allow for 
learner-lesson interaction through learner overt response and program-provided 
feedback. Importantiy, the number of answer options is fioiig, and individualization 
via branching is possible based upon the learners' correct and incorrect responses. 
Additionally, multiple-choice tests can be delivered and marked by computer thus 
increasing the efficiency of the instructional process. 

However, if multiple-choice items are used during instruction, it would be 
inappropriate to use the exact same items for post-lesson assessment. Such a test 
would only measure rote learning. In order to take advantage of the convenience of 
multiple-choice items, one often-used procedure is to utilize multiple-choice items 
during the lesson and then use parallel constructed-response items on the posttest. 
This method is straightforward and effective. One minor problem is that 
computer-maiking of constructed-rcsponse type items (i.e. fill in the blank) is 
sometimes problematic. Learners may know the correct answer but may miss an item 
due to syntax, spelling, or other unexpected variables. Often constructed-response 
posttests must be remarked by hand 

The purpose of this study is to describe a method for adapting multiple-choice 
items so that parallel forms of the same item may be used during the lesson for 
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instruction and during the posttest for assessment Data from three studies are shown 
to support this item modification method. 

Transfcxrnational Analysis 
The term transformational analysis (Bormuth, 1970) applies to the process of 
generating test items from instructional text (see Figure 1). Anderson (1972) 
described two methods for develq)ing test items from reading passages in order to 
asses comprehension. These methods include paraphrasing and transforming. 
Paraphrasing involves changing the words from the reading passage by replacing 
words with equivalent words (i.e. semantic substitute), while maintaining the meaning 
of the reading passage. Transforming , in this instance, involves reversing the original 
order of the sentence subject and predicate elements The noun phrase in the subject 
of the sentence becomes the correct answer option of the parallel posttest item; and the 
noun phrase in the predicate becomes the stem of the parallel posttest item (i.e. passive 
transfomi). 



Reading Passages 



Posttest Items 



transfOTm 
processes 



subject deletion items 

echo items 

tag items 

roteiiems 

transform items 

senumtic substitute items 

...others 



Figure 1 . Modifying posttest items from reading passages (Bormuth, 1970). 
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These two methods arc not mutually exclusive. An item might be 
not-transftnmcd and not-paraphrased (i.e. verbatim, also termed rote), transformed 
and not-paraphrased (i.e. transformed), not-transformed and paraphrased (i.e. 
paraphrased), and compound (i.e. transformed and paraphrased). Thus four parallel 
multiple-choice items can be aeated by crossing these two procedures. A very simple 
example of a lesson sentence and four parallel items based on this sentence arc shown 
in Table 1. 

Table 1 

Four parallel items derived from a sentence taken from a lesson. 



Sentence from a lesson : Littie Rock is the Capital City of Arkansas. 



Verbatim : Littie Rock is (a) die Capital City of Arkansas (b) tiie Capital City of 
Tennessee (c) the Capital City of Missouri. 

Transformed : The Capital City of Aricansas is (a) Littie Rock (b) Memphis (c) 
Arkadelphia. 

Paraphrased : Littie Rock is the official seat of government in (a) Arkansas (b) 
Tennessee (c) Missouri. 

Transformed-Paraphrased : The official seat of government of Arkansas is in (a) Little 
Rock (b) Memphis (c) Aikadelphia. 



Bormuth, Manning, Carr, and Pearson (1970) used posttest items tiiat were 
verbatim from tiie rcading passage, paraphrased from tiie reading passage, 
transformed from the rcading passage, and both paraphrased and transformed from the 
rcading passage. These four conditions arc displayed in a two by two matrix (see 
Figure 2) with tiie following scores on tiie immediate posttest: verbatim, 77%; 
transformed, 71%; paraphrased, 69%; and transformed-paraphrased 67%. Verbatim 
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(i.e. not pan^hrased and not transformed) items obtained the highest scores with the 
transffflined and paraphrased items next The combination of transform and 
parq)hrase methods produced the most difficult items. 



^sformed Transfgrmgd 



Paraphrased 
Paraphrased 



77% 


71% 


69% 


67% 



Figure 2 . Scores for the four types of items used by Bormuth et al. (1970). 



Bormuth (1970) observed, "It is a conunonly observed phenomenon in test 
writing that altering the form of an item has some effect on its difficulty" (p. 15). 
Applying these two methods appears to impact item difficulty and thus posttest 
performance for each type of item in a logical way (Davey, 1988). Anderson ( 1970) 
argues that these items provide a measure of comprehension of lesson materials. 

This paper sugg ests that paraphrasing and transforming can be applied to lesson 
items as well as to reading passages (see Figure 3). 

Lesson Questions Posttest Items 



Verbatim 

Paraphrased 

Transformed 

Paraphrased & Transformed 



Figure 3 . Modifying posttest items from lesson items. 
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This would allow for the development of posttest items that parallel lesson items and 
provide a measure of comprehension. If the transform and paraphrase methods 
described above are employed, again four types of parallel posttest items will result. 
What are the effects on posttest performance of these different item types? The results 
of three studies are described to support this methodology. 

Study 1 

A sample of 32 "at risk" eleventh-grade students completed a computer-based 
social studies lesson composed of 32 four-option multiple-choice items with feedback 
which included four associated 350-word reading passages. The posttest consisted of 
32 parallel four-option multiple-choice items, half were verbatim to the lesson items 
and half were paraphrased from the lesson items. The results for these paraphrase 
posttest items were similar (i.e. compare Figure 4 to Figure 2) to those reported by 
Bormuth et al. (1970). Transformed questions were not used. 



Not 
Transfq 



Transformed 



Not 

Paraphrased 
Paraphrased 



n.5% 


n/a 


59.8% 


n/a 



Figure 4 . Posttest Results for Study 1 . 



Study 2 

A sample of 36 "average" tenth-grade students completed a workbook 
containing four natural-science reading passages and 40 related multiple-choice items 
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provided with different forms of feedback. The posttest included all four types of 
modified items, both transformed and paraphrased. The results for each question type 
are shown in Figure 5. 



Not 

Transformed Transformed 



Not 

Paraphrased 
Paraphrased 



47.9% 


39.2% 


40.8% 


30.0% 



Fi gures . Posttest Results for Study 2. 



Study 3 

A group of 103 "at risk" eleventh-grade students completed a computer-based 
science lesson consisting of 40 four-option multiple-choice items with feedback. 
Some also received 4 associated 350-word reading passages. The posttest consisted 
of 40 four-option multiple-choice items that were parallel to the lesson items. This 
study used the same lesson and posttest items as Study 2 above. These posttest items 
consisted of all four categories that can occur by using the transforming and 
paraphrasing methods. The results (see Figure 6) again were similar to those reported 
byBormuth et al. (1970). 



Transformed Transformed 



Paraphrased 
Paraphrased 



80.4% 


47.7% 


46.3% 


40.4% 



EigUifi_6. Posttest Results for Study 3. 
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Discussion 

Transfcnining and paraphrasing lesson text to form multiple-choice posttest 
items is not a new idea. Applying this same method to lesson items rather than lesson 
text allows designers to utilize multiple-choice items both fcx- instruction during the 
lesson and for assessment during the posttest. The test items produced will obviously 
parallel the lesson items one-toone in terms of content (i.e. instructional objectives) 
and appear to create a systematic range of item difficulties. These results are viewed 
as evidence that many of the cognitive skills underlying comprehension "may be 
hierarchically related" (Bormuth et al., 1970). 
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Note: 

The data reported in Study 1 was part of a previously reported study titled "Two 
immediate feedback forms with two conditicMis of contextualization" in the 
Journal of Computer-Based Instruction , in press. The data from Studies 2 and 3 
were part of a dissertation completed at Memphis State University tided "An 
experimental study comparing three forms of feedback across two conditions of 
acquisition". 
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